Enabling differential performance benchmarking #4667
Conversation
tests/regression/src/lib.rs
Outdated
| let suffix = get_test_suffix(); | ||
| if !is_running_under_valgrind() { | ||
| let ctrl = InstrumentationControl; | ||
| test_body(&ctrl) | ||
| if is_diff_mode() { | ||
| run_diff_test(test_name); | ||
| Ok(()) | ||
| } else { | ||
| let ctrl = InstrumentationControl; | ||
| test_body(&ctrl) | ||
| } | ||
| } else { | ||
| run_valgrind_test(test_name); | ||
| run_valgrind_test(test_name, &suffix); | ||
| Ok(()) |
There was a problem hiding this comment.
These changes are a great example of why it's a good idea to consolidate/abstract how the tests are run from the harness definitions themselves. If we didn't have this abstraction, you'd have to go update every harness (and hopefully you consistently applied the change :))
|
In the newest revision, I have included changes to the way |
Resolved issues:
N/A
Description of changes:
This PR builds on #4649 to introduce differential benchmarking. Specifically, it adds a suffix flag to toggle between versions of s2n-tls to compare them and a diff_mode flag to control scalar vs differential performance analysis. The actual diff functionality relies on
cg_annotate --diffwhich diffs two cachegrind output files to pinpoint the sources of performance difference.Call-outs:
target/perf_outputsbased on the version attributed to them,curr,prev, ordiff. This makes it easier to access the output files instead of having to skim file names between test versions, taking into consideration the workflow of embedding them into CI and downloading them through the GitHub Actions workflow.Testing:
This addition does not change any existing functionality. The benchmarking functionality was tested by following the developer workflow outlined in the README for differential performance.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.